Scalable Self-Organizing Overlays

Recent developments including peer-to-peer networks and application-level multicast have generated interest in overlays, and by extension in algorithms for automatically maintaining overlay structures. In this paper, we present a self-organizing protocol for forming and managing overlays that scales to at least tens of thousands of nodes. Previous algorithms target small scale overlays and do not scale well in terms of protocol overhead with the number of nodes. Our key contribution is to show how to apply the concept of hierarchy to overlay management to achieve scalability, without degrading the quality of the resulting overlay. We provide simulation results that show our hierarchical overlays incur a low delay penalty, make effective use of network bandwidth, and handle failures gracefully. We also demonstrate that there are significant advantages in using a selforganizing overlay as compared to the ad hoc techniques currently being used in peer-to-peer networking.